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ABSTRACT 



This paper describes each of eight current interrelated 
federally funded research projects conducted by the Research and Training 
Center for Children's Mental Health at the University of South Florida. For 
each project, information is provided on the project's background, purpose, 
methodology, and anticipated results. Project titles and principal 
investigators are: (1) "Developing a Framework for Assessing the Impact of 

Policies on Children with Emotional Disturbances and Their Families" (Robert 
Friedman) ; (2) "The Impact of Policy on Collaboration in Systems of Care for 

Children with Emotional Disturbance and Their Families" (Mary E. Evans, 
Barbara Morrison-Rodriguez, and Cheryl Cockram) ; (3) "The Urban School and 

Community Study: Effects of Systems Reform on Students with Emotional and 
Behavioral Disabilities and Their Families" (Krista Kutash and Albert J. 
Duchnowski) ; (4) "Community- Based Theories of Change" (Mario Hernandez and 

Sharon Hodges) ; (5) "Health Care Tracking Project" (Mary Armstrong, Beth 

Stroul, and Sheila Pires) ; (6) "Impact of Managed Care and System of Care 

Policies on Service Access for Children with Culturally Diverse Backgrounds" 
(Paul Greenbaum and Eric C. Brown) ; (7) "The Child and Family Experience of 

the Mental Health System" (Katherine Lazear) ; and (8) "Development and 
Validation of an Instrument To Measure Interagency Coordination and 
Collaboration" (Paul Greenbaum and Eric C. Brown) . (DB) 
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An Overview of the 

5-year Research Agenda 

for the Research & Training Center 

for Children's Mental Health 



Introduction 

The Research and Training Center for Children’s Mental Health at 
the University of South Florida began a new set of eight interrelated 
research projects in October 1999 with funding from NIDRR and 
the Center for Mental Health Services. This summary briefly 
describes each of eight research projects. The projects focus on the 
impact of policy on system change and collaboration, community- 
based theories of change, school-wide reform and mental health 
policies in urban schools, health care reform, the impact of managed 
care and system of care policies on service access for children with 
culturally diverse backgrounds, the perceptions of family members of 
the service delivery system, and the development and validation of an 
instrument to measure interagency coordination and collaboration. 
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Study 1: Developing a Framework for Assessing the Impact of 
Policies on Children with Emotional Disturbances and Their Families 



Robert Friedman 

Background This study examines the relationship between policy development and 
implementation in children’s mental health and actual changes at the system and practice level. While 
the establishment of policy may be an important step in building effective systems of care, unless the 
policy is carefully designed to address the major issues, and unless there is a strong implementation 
plan that is carried out well, the impact of the policy itself may be minimal. Despite this, there has 
been very little systematic investigation of the impact of policy on system development, and on 
outcomes for children and families. As a result, the needed information is not yet available to guide 
policy makers and administrators in determining how to develop and implement policy that will 
positively impact children with emotional disturbances and their families. 



Purpose. The purpose of this project is to develop an analytic framework, based on the best available 
research, to guide thinking about the relationship between policy development, policy implementation, 
systems of care, and outcomes in childrens mental health. Through development of the framework, the 
Center investigators seek to identify key questions that need to be addressed both by researchers and 
individuals interested and involved in the policy development and implementation process. 

Methodology, The Center has a specific focus on important policies affecting children with 
emotional disorders and their families. In particular, the Center is interested in policies designed to 
create effective community-based systems of care that involve collaboration between different service 
sectors, policies that help establish effective school programs, and financing policies designed to 
contain costs while still providing access to needed services. The framework that will be established as 
part of this project will be applied in relation to these policies, (see Figure 1). 

The framework begins by examining the context for the identification of an issue to be addressed 
through public policy. Both the political and economic context will be examined. In addition, the 
nature of the problem to be addressed, and the population of concern, also constitute an important 
part of the context that will be studied in this project. 

best copy available 
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The framework will then examine a set of four factors related to the content of policies themselves. 
These variables are: the extent to which the values, principles, and content of the policy are consistent 
with the values, principles and content of community based systems of care; the extent to which 
related policies in other sectors are based on similar values, principles, and concepts; the clarity of the 
policy in expressing actions to be taken; and the clarity and validity of the causal theory upon which 
the policy is based. 

The framework will also examine five factors related to the implementation of the policy. These 
variables are: the resources that are dedicated to addressing the problem; the accountability procedures 
that are established to determine if the policy is achieving its purposes; the policy instruments or 
strategies that are used as the basis for the policy; the broad overall belief system about how change is 
made that is used to develop the policy; and the assignment of responsibility for implementation of 
the policy. 

The framework recognizes that policies have both short-term and long-term outcomes, and that it 
is important to examine both. It also recognizes that policies often have unintended consequences 
which may be either positive or negative. As part of the framework, these outcomes will be studied, 
and the manner in which outcome information is then used to modify either the policy or the 
implementation strategy will also be examined. 

It is expected that policies that are consistent with system of care values, principles, and concepts 
will result in higher levels of collaboration, that policies that have been developed in a manner that 
allows discretion in the community will be more effective than policies that are more top-down in 
their approach, that policies that have used capacity-building and system-change strategies will be 
more effective than policies that depend on mandates, and that policies that include accountability 
procedures with clear expectations, and which also include the collection of practical data for 
communities will be more effective. 

The initial work in developing the analytic framework took part during the process of preparing 
the Center grant application. The preliminary framework was developed and included as part of the 
grant proposal, both to provide cohesiveness and direction for the series of studies to be done by the 
Center, and to provide reviewers with the best information about the framework. In addition, this 
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framework has been discussed with the Center's Board of Advisors, and sent to 21 individuals around 
the country for review and comment. These reviewers were selected to include individuals from diverse 
backgrounds and with varied roles in the system, including policy analysts, administrators, parents, 
researchers, policy makers and practitioners. Reviewers represent federal, state and local levels. The 
framework will be reviewed a second time with Center staff in March 2002, at the mid-year of the 
five-year grant cycle. In March 2004, Center staff will hold a meeting to review the framework again, 
in the context of the results from the Center projects. In the interim, Center staff will continue to 
disseminate the version of the framework that emerges in November 2000, after comments from the 
reviewers have been addressed. 

Anticipated Results. This framework is intended to provide a direction for the other studies to be 
undertaken by the Center. The framework will be reviewed and potentially modified at several points 
during the grant period, based primarily on the results of the research to be conducted. 

Study 2: The Impact of Policy on Collaboration in Systems of Care for 
Children with Emotional Disturbance and Their Families 

Mary E. Evans, Barbara Morrison-Rodriguez & Cheryl Cockram 

Background . The development of systems of care to serve children with serious emotional disorder 
(SED) and their families is not new, having been supported by the National Institute of Mental 
Health’s Child and Adolescent Service System Program (CASSP) initiated in the 1980s. The 
components of this system, its core values and guiding principles have been articulated well by Stroul 
and Friedman (1986) and continue to be as relevant today as when they were published. As 
communities have begun to initiate such systems of care in their local areas, however, there has been 
renewed interest in examining collaborations within systems of care. The purpose of this study, 
developed as part of the Research and Training Center for Children’s Mental Health grant application 
to its funding agencies, is to determine the relationship between policy development, policy 
implementation, systems of care and outcomes in childrens mental health. Additionally, this study will 
examine the relationships among systems of care, legislative policy and the development of 
collaborative systems. 

Policies related to the development of systems of care can generally be viewed as having used a 
forward or backward approach. A forward approach originates at the level of federal or state policy 
makers and is passed down to lower levels of government, while a backward approach originates at the 
level of local communities, families, Native American tribes or similar groups of concerned citizens. 
Based on this view of policy, this study has the following hypotheses: 

• Policies that have been developed and implemented using a backward mapping approach will be 
more successful in producing effective collaborative systems at the community level than policies 
that have been developed using a forward mapping approach. 

• True collaboration is more likely to occur when policies reflect system of care principles, have 
clearly articulated goals and target populations and provide adequate human and financial 
resources. 

Purpose . The purpose of this study is to test the analytic framework developed in Study 1 to guide 
thinking about the relationship between policy development, policy implementation, systems of care, 
and outcomes in children’s mental health. Specifically, this study will examine the relationships among 
systems of care, legislative policy and the development of collaborative systems at the local level. 

Methodology . This study of collaboration, currently early in its implementation, has two phases. The 
first phase consists of a national survey of key informants regarding the policy instruments that have been 
used in their state to develop systems of care. These key informants are the primary persons identified by 
the National Association of State Mental Health Program Directors or their designees. Policy instruments 
may include such things as laws, regulations, negative sanctions, inducements, resource allocation and 
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other means to creating capacity to serve children and families. Following the receipt and review of these 
materials from the program or policy staff at the state level, a matrix of the types of systems of care will be 
developed. This matrix will include information about age of the system, the policy instruments used to 
develop the system, the levels of government involved and other salient variables. Systems will be 
clustered on the basis of these variables and approximately four sites, representing different approaches to 
implementing systems of care, will be selected for case studies. The intensive case studies, which represent 
phase two of the study, will utilize site visits to obtain information about collaboration with the systems. 

Three primary approaches are used to study the relationship between policy instruments related to 
the development of systems of care and the resulting collaboration within systems. First, the project 
uses a case study approach involving brief, intensive Fieldwork at four sites. Secondly, the fieldwork 
uses a backward mapping approach (Elmore, 1983) to examine collaboration, beginning at the local 
community level and working upward toward the state level. Thirdly, mixed methods, both 
quantitative and qualitative, are used. The mixed methods include interviews with families, providers 
and policy makers, record reviews, observation of group meetings, and examination of written policies 
and procedures. In addition to these methods, selected individuals representing different 
constituencies at the four sites that will be identified for intensive case studies will be asked to 
complete an instrument designed to measure collaboration within systems of care. Other faculty 
members, Dr. Paul Greenbaum and Eric Brown, at the Research and Training Center for Childrens 
Mental Health, are developing this instrument. The types of factors that the study team will be 
looking for during the site visits are those factors that facilitate or inhibit collaboration (See Figure 1). 

We believe that facilitating and inhibiting factors fall into three major categories: attitudinal, 
behavioral, and structural or organizational. It is proposed that the intersection of facilitating factors in 
these three domains results in collaboration. 



Figure 1 

Facilitating and Inhibiting Factors to Collaboration 
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Anticipated Results . We have identified several desired outcomes for this study. First, this study 
will provide an empirical test of the policy framework that was developed in the foundational study for 
the Research and Training Center for Childrens Mental Health grant application. Secondly, we 
anticipate that the study will enrich the field s understanding of the process between state level policy 
and local implementation of systems of care. Thirdly, we anticipate that the study will assist in 
identifying critical mediating and contextual variables that are important in developing collaboration 
within systems of care. Fourthly, the study may provide guidance for future efforts to develop more 
effective policies and collaborations. 
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Study 3: The Urban School and Community Study: Effects of 
Systems Reform on Students with Emotional and Behavioral 
Disabilities and their Families. 



Krista Kutash & Albert J. Duchnowski 

Background. Several recent studies indicate that the public school system is the major provider of 
mental health services for children who have emotional and behavioral problems. The current school 
reform movement is intended to improve outcomes for all students including students with 
disabilities. However, there is very little information about the effects of school reform on children 
who have emotional and behavioral disabilities and their families. Building on previous research by the 
Center on school reform in suburban and rural communities, the current study will investigate school 
reform in urban communities that service diverse populations of children and families. 

Purpose . The purpose of this study is to describe the various models of school reform/ 
improvement operating in urban areas that serve diverse populations of children and families. 
Specifically, the study will examine the relationship between these reform activities and selected 
measures of student functioning for those students in special education settings due to emotional and 
behavioral disabilities as well as the services they are receiving. 



Methodology . Essentially, we will conduct a study employing a quasi-experimentai design in which 
student functioning, service accessibility, program characteristics, and family satisfaction in schools 
rated very successful in reform will be compared to schools that are not as successful. In this way we 
will more clearly be able to examine the policies and procedures driving system reform and the effect 
on children who have emotional disturbances and who are from ethnically and racially diverse 
backgrounds. 



Over the five-year period of the study, 20 schools will be selected to participate. Staff from these 
schools will participate in a series of interviews about reform and improvement efforts operating at 
their schools. From these interviews, varying aspects and levels of reform can be captured and 
examined in terms of their relationship to student functioning. Additionally, information will also be 
collected from parents describing their perceptions of the school reform process and their satisfaction 
with their child’s program and levels of functioning. 



A structured interview protocol has been developed for administration to school and community 
staff in order to establish a “reform score.” Scoring the interview has proven to be reliable, yielding an 
intraclass correlation among raters of .86. The current efforts will also produce data evaluating the 



13th Annual Conference Proceedings-A System of Care for Children s Mental Health: Expanding the Research Base- 11 



Friedman, Evans, Morrison-Rodriguez. Kutash, Duchnowski, Armstrong, Fires, Stroul, Greenbaum, Brown & Lazear 



validity of the instrument. Several measures of academic achievement and emotional functioning will 
be administered to the student participants and their families, including academic achievement, school 
attendance, suspension rates, and levels of emotional and behavioral impairment. We will also examine 
variables such as attendance, discipline referrals, time spent in special education classes versus time 
with non-handicapped peers, and general demographic characteristics. Finally, there will be a 
description of the extent of related services received by each participant, the source of these services, 
and funding mechanisms for these services. 

Anticipated Results. An anticipated result of this study is the refinement of a method to assess 
school reform that may be used in both urban and rural communities. Several schools will participate 
each year of the study affording periodic dissemination of various reform and special education and 
delivery models. Finally, data from this study will further clarify the relationship between school 
reform/improvement and the functioning levels of children who have emotional and behavioral 
disabilities and their families. 

Study 4: Community-Based Theories of Change 

Mario Hernandez and Sharon Hodges 

Background. The Center believes that without an explicit understanding of why services are 
expected to affect change among children and families, any outcomes accomplished cannot be easily 
interpreted or achieved. 

Understanding the assumptions that underlie local-level policies and service planning, referred to in 
this study as community-based theories of change, involves the processes through which service-related 
policy is effectively conceptualized and operationalized and considers how these processes support the 
successful implementation of services as intended. This study will investigate the organizational processes 
that support and facilitate the relationship between the implementation of services and the 
conceptualization and operationalization of local-level policy and service delivery strategies. 

Purpose. The purpose of this study is to investigate the organizational processes that support the 
implementation of local-level policy, to investigate how local evaluation efforts support policy 
implementation and incremental correction when needed; and to investigate whether clarity in the 
conceptualization and operationalization of local policy is useful in building consensus and enhancing 
cultural competence. 

Methodology. A multi-site case will be conducted in three phases. During Phase I (years 1 and 2), 
system and program sites will be selected based upon their ability to successfully implement local 
service policy as reflected in a widely held theory of change. During Phase II (years 3 and 4), sites that 
both confirm and disconfirm identified patterns of successful local policy implementation will be 
investigated. During Phase III (year 5), results of the study will be shared through technical assistance 
to program and system level sites as well as workshops, presentations, and a published monograph. 
Phone interviews were conducted with key informants at ten potential sites, and site selection visits 
were conducted at four sites. A data summary tool was developed and used to analyze site selection 
data and provide feedback to each site. 

Anticipated Results. The goals of this project are to gather information on how systems and 
programs accomplish conceptualization, operationalization, and implementation of community-based 
service policies; to explore how systems and programs transfer policy agendas across stakeholders in 
local organizations, and how local service strategies are transferred, diffused, and sustained over time, 
to understand how clear conceptualization and operationalization of local childrens mental health 
policy are useful in building consensus and enhancing cultural competence within a system or 
program; to provide a tool that will facilitate the ability of community-based stakeholders to reach 
consensus regarding their strategies and outcomes. 
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It is hoped that this study will foster additional research into the processes through which service- 
related policy is implemented within a variety of organizational and environmental contexts (e.g., 
managed care organizations, profit and non-profit organizations, welfare reform, urban and rural 
communities). In addition, it is hoped that this research will lead to testing the impact of different 
models of service delivery with the ultimate goal of learning how these models impact changes in 
functional outcomes of the children and families served. 

Study 5: Health Care Tracking Project 

Mary Armstrong, Beth Strou! & Sheila Pires 

Background. The Health Care Tracking Project (Tracking Project) is being undertaken during a 
period of rapid change in public sector health and human service systems. States and, increasingly, 
local governments are applying managed care technologies to the delivery of mental health and 
substance abuse services for children, adolescents and their families within Medicaid, mental health, 
substance abuse, child welfare and State Childrens health Insurance (SCHIP) programs. The Tracking 
Project is the only national study focusing specifically on the impact of these public sector managed 
care reforms on children and adolescents with behavioral health disorders and their families. 

Purpose . The Tracking Project was initiated in 1995 to track and analyze state and local managed 
care initiatives as they affect children and adolescents with emotional and substance abuse disorders 
and their families. 

Methodology. The Tracking Project involves three major components — surveys of all states, for 
which quantitative, descriptive statistics will be used; in-depth qualitative case studies of state and 
community sites with promising mental health financing and managed care approaches; and a 
consensus conference held during the fourth year of the grant. The survey of all states is being 
conducted during the first year, and will be repeated in the fourth year of the project. The in-depth 
case studies of state and community sites will be conducted during years two and three of the project. 
The fifth year of the project will be used to integrate the results of the consensus conference and the 
final survey of states. 

Anticipated Results. Over the next five years, the Tracking Project will continue to conduct all- 
state surveys to track developments in public sector managed care affecting children and adolescents 
with behavioral health disorders and their families. In addition, the Project will engage in a study of 
promising approaches to address the needs of this population within the context of publicly financed 
managed care, and will convene a Consensus Conference to compare findings from the Tracking 
Project to findings from related studies, with a view toward developing consensus recommendations 
about the policy and implementation strategies that most effectively serve this population within the 
current environment. 

Study 6: Impact of Managed Care and System of Care Policies on 
Service Access for Children with Culturally Diverse Backgrounds 

Paul Greenbaum & Eric C. Brown 

Background. Both managed care and system of care have targeted access to services as a primary 
focus of their policy initiatives. However, few studies have examined the effects these policies have had 
on children from culturally diverse backgrounds. Analyses of existing data, which have been collected 
to study the broad impact of these policies, have been used to examine if these policies have affected 
access to mental health services, specifically, for children of color. 

Purpose. The major purpose of this project has been to explore the impact of two recent policy 
initiatives, managed care and system of care, on access to services for children of color. A second 
purpose of this study has been to test the cultural equivalence of instruments that assess mental health 
service need. Meeting this assumption, referred to as “measurement invariance, ensures that scores 
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from different groups are on the same metric, and therefore, are measuring the same construct and are 
directly comparable. Without cultural equivalence, meaningful comparisons of service need and 
derived measures of service access between children from different cultures are not possible. 

Developing cultural-fair assessment measures has been cited as a priority for developing policies that 
are culturally competent. Only recently have the necessary analytic tools been developed that permit a 
systematic assessment of measurement invariance for instruments. 

Methodology. During the first year of the project, using data from the Florida Health Services 
Survey, two types of managed care systems (i.e., HMO carve-in, carve-out) versus a traditional fee-for- 
service system for equal access to services were examined. Additionally, the culture fairness of the 
Pediatric Symptom Checklist (Jellinek, Murphy & Burns, 1986) was tested. This year, the focus of the 
project has been to continue to look at the issues of equal access and culture-fairness of assessment 
instruments within the system of care. Data from Macro longitudinal comparison study of the 
Comprehensive Community Mental Health Services for Children and Their Families were used for 
analysis. As part of the evaluation, three system of care demonstration sites were compared with three 
matched sites of traditional mental health systems, so that differences between system of care and 
traditional models could be examined. Service use and service cost including service type, units of 
service, length of service, and service unit cost have been collected from the management information 
system (MIS) available at each site. 

Anticipated Results, Testing the cultural equivalence of the CBCL/4-18 (Achenbach & 

Edelbrock, 1991). and the CAFAS instruments used to measure service need has been completed. 
Testing differences by system of care versus traditional care for service need has also been completed. 
Data for service use (e.g., type and amount), and service costs are in the final data-cleaning phase and 
the results from Macro are expected soon. 

References 

Achenbach, T. M., & Edelbrock, C. (1991). Child Behavior Checklist/4-18. Burlington: University 
of Vermont, Department of Psychiatry. 
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Study 7 : The Child and Family Experience of the Mental Health System 

Katherine Lazear 

Background. This study consists of two related projects that capture the perspective of families as 
they respond to current policies and practice. Project A, the Supplemental Security Income (SSI) 
Impact Study examines the potential impact of the Welfare Reform SSI federal policy changes in 
eligibility criteria for SSI disability benefits, the new strict definition of “disability,” and the mandated 
reviews of children enrolled. Project B is the Child and Family Experience of the Mental Health 
System. This study responds to research findings that many children need services, but only a small 
percentage receive them; it is typically at least two years before parents obtain appropriate help; and, 
families leave treatment before completing it at a high rate. While the challenge for the system is to 
demonstrate that it can reach families in need and retain them in services, there is a lack of data on 
child and parent perspective of the treatment experience. 

Purpose . This study is designed to meet a need for family perspectives about the general impact of 
policy and services, as well as the specific aspects that are the most and least helpful. 

Methodology. Project A, The SSI Family Impact Study is now in the final analysis phase. The 
focus groups and in-person interviews over the twenty-two month with parents have been completed. 
Thirty of the original forty families remained in the study. Available data from the study includes: 
families’ socio-demographic and socio-economic profiles; description of families’ perceptions of 

9 



14 - Research and Training Center for Childrens Mental Health -Tampa, FL - 2001 



— Research Agenda of the Research and Training Center for Childrens Mental Health 



process for original determination of eligibility; primaty care givers’ reactions to changes in eligibility; 
decision-making history in response to re-determination of benefits; and, the role of informal and 
formal supports in families’ coping with changes. Project B utilizes a longitudinal design with frequent 
interviews with parents and children, and a comparison of high frequency and low frequency contact 
groups. 

Anticipated Restdts. This study will increase the field’s understanding of the experience of 
children and their families around help-seeking and treatment continuation. It will also help to 
identify factors in the treatment experience that parents perceive to be helpful. And, it will strengthen 
and complement research on empirically supported treatment through qualitative information from 
those who experience treatment. 

Study 8: Development and Validation of an Instrument to Measure 
Interagency Coordination and Collaboration 

Paul Greenbaum, & Eric C. Brown 

Background \ Interagency collaboration has been a key concept in developing systems of care. 
Successful collaboration involves having coherent policies that coordinate resources, decision making, 
and problem solving. Such policies have been viewed as a primary mechanism for improving children’s 
chances of receiving the most appropriate set of needed services. However, few instruments exist that 
measure interagency collaboration in children’s mental health and little is known about how these 
measures perform. 

Purpose. The purpose of this study is to develop a comprehensive and psychometrically sound 
survey instrument that measures interagency collaboration. Based on the developed instrument, this 
study will evaluate the amount of variability in interagency collaboration that is associated with 
within-system factors (i.e., agreement among staff members) and between-system factors (i.e., between 
agencies or systems). To support the work of researchers, this study will provide data on the optimum 
number of items needed to adequately measure the construct of interagency collaboration, as well as 
the number of staff-respondents per system and systems needed to adequately represent interagency 
collaboration and its relation to service outcomes in statistical designs. 

Methodology. The study is being conducted in two phases: (a) instrument development and (b) 
data collection and analysis. During the first phase of the project a systematic set of procedures has 
been used to: (a) define and operationalize the construct of interagency collaboration, (b) identify 
items that capture the elements that constitute this construct, and (c) pilot the new instrument for 
final revisions. During the first year of the study, a comprehensive literature review based on 
computerized bibliographic search procedures was conducted to identify publications, theoretical 
statements, and instruments related to interagency collaboration. Based on the literature review and 
interviews conducted in Year 1, a 3-factor model (i.e., beliefs/values, collaborative activities, knowledge 
of other organizations) has been selected to model the construct of interorganizational collaboration. 
Subsequently, from the item bank of more than 80 potential items, 40 items were selected as 
operationalizing the three-factor model of interorganizational collaboration. A review panel of 20 
experts is currently evaluating the preliminary instrument. 

Anticipated Results. Knowledge developed through this study will allow researchers to create a 
design with adequate power to study the relations between interagency collaboration, service 
coordination, and children’s outcomes. 
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